A database is a structure for storing and relating data within e.g. a computer system. Different database architectures exist depending on the intended usage. The primary use for general purpose databases is to manage and facilitate data entry and retrieval in relation to the relevant application. A recent trend has been the emergence of specialized database architectures optimized to work with specific application domains.
Complex event processing (CEP) is a technology for low-latency filtering, correlating, aggregating and/or computing on real-world event data, e.g. financial data. Such data is usually generated at high frequencies and so needs to be saved in an appropriate database to allow it to be evaluated, whether in real time, or at a later stage. Several specialized database products have emerged which attempt to store such data, which is generated in quantities that normally overwhelm general purpose databases.
The following products are available for use in CEP applications, and provide different functionalities for manipulating CEP data.
ProductDescriptionTechnologyVhayuHigh performance proprietary databaseProprietary,velocityoptimized to work with high-frequencynon-relationalfinancial market datain-memorydatabaseKX systemsHigh performance database to monitorOptimized,KDB+real-time events and detect and reportcolumn-basedfaults for data-intensive applicationsdatabaseStreamBaseEvent processing platform which allowsIntegratedfor development of applications that querydevelopmentand analyze high-volume real-time dataenvironmentstreamsalong withspecializedcompiler
These products aim to provide improvement of both underlying database technologies and processing capabilities. However, data storage and querying or retrieval of the data is still carried out according to conventional processes. While these databases are well-suited to performing traditional transaction-oriented operations, they do not provide an efficient means for allowing large amounts of contiguous data to be accessed and/or evaluated, other than standard querying methods.
Such requests for large amounts of contiguous data are relevant to the provision of descriptive statistics where the importance of individual records is less than the overall description. Descriptive statistics are now becoming increasingly important especially for high-frequency high-volume data applications.
The process of evaluating large contiguous datasets is central to responding to statistical descriptive data requests.
The financial services community consists of data providers and clients. Data providers deal with both large institutional clients (e.g. banks) and smaller clients (e.g. retail investors). Dealing with the larger clients is either done directly or through 3rd party vendors (e.g. Vhayu) to provide them with all market data in order to allow for the construction of sophisticated and accurate statistical variables. However, at present this is not possible with smaller clients due to costs associated with the large bandwidth and computational requirement needed for delivering the complete market feed. Therefore, smaller clients are only provided with market snapshots or summaries, which only allow for variable approximations.